<!DOCTYPE html>


<html lang="zh-CN">


<head>
  <meta charset="utf-8" />
    
  <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1" />
  <title>
    Spring Cloud 应用在 Kubernetes 上的最佳实践 — 高可用 混沌工程.md |  
  </title>
  <meta name="generator" content="hexo-theme-ayer">
  
  <link rel="shortcut icon" href="/favicon.ico" />
  
  
<link rel="stylesheet" href="/dist/main.css">

  
<link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/Shen-Yu/cdn/css/remixicon.min.css">

  
<link rel="stylesheet" href="/css/custom.css">

  
  
<script src="https://cdn.jsdelivr.net/npm/pace-js@1.0.2/pace.min.js"></script>

  
  

  

</head>

</html>

<body>
  <div id="app">
    
      
    <main class="content on">
      <section class="outer">
  <article
  id="post-k8s/Spring Cloud 应用在 Kubernetes 上的最佳实践 — 高可用 混沌工程"
  class="article article-type-post"
  itemscope
  itemprop="blogPost"
  data-scroll-reveal
>
  <div class="article-inner">
    
    <header class="article-header">
       
<h1 class="article-title sea-center" style="border-left:0" itemprop="name">
  Spring Cloud 应用在 Kubernetes 上的最佳实践 — 高可用 混沌工程.md
</h1>
 

    </header>
     
    <div class="article-meta">
      <a href="/2020/11/11/k8s/Spring%20Cloud%20%E5%BA%94%E7%94%A8%E5%9C%A8%20Kubernetes%20%E4%B8%8A%E7%9A%84%E6%9C%80%E4%BD%B3%E5%AE%9E%E8%B7%B5%20%E2%80%94%20%E9%AB%98%E5%8F%AF%E7%94%A8%20%E6%B7%B7%E6%B2%8C%E5%B7%A5%E7%A8%8B/" class="article-date">
  <time datetime="2020-11-10T16:00:00.000Z" itemprop="datePublished">2020-11-11</time>
</a> 
  <div class="article-category">
    <a class="article-category-link" href="/categories/k8s/">k8s</a>
  </div>
  
<div class="word_count">
    <span class="post-time">
        <span class="post-meta-item-icon">
            <i class="ri-quill-pen-line"></i>
            <span class="post-meta-item-text"> 字数统计:</span>
            <span class="post-count">2k</span>
        </span>
    </span>

    <span class="post-time">
        &nbsp; | &nbsp;
        <span class="post-meta-item-icon">
            <i class="ri-book-open-line"></i>
            <span class="post-meta-item-text"> 阅读时长≈</span>
            <span class="post-count">6 分钟</span>
        </span>
    </span>
</div>
 
    </div>
      
    <div class="tocbot"></div>




  
    <div class="article-entry" itemprop="articleBody">
       
  <h1 id="Spring-Cloud-应用在-Kubernetes-上的最佳实践-—-高可用（混沌工程）"><a href="#Spring-Cloud-应用在-Kubernetes-上的最佳实践-—-高可用（混沌工程）" class="headerlink" title="Spring Cloud 应用在 Kubernetes 上的最佳实践 — 高可用（混沌工程）"></a>Spring Cloud 应用在 Kubernetes 上的最佳实践 — 高可用（混沌工程）</h1><p><strong>导读：</strong>从上篇开始，我们进入到了高可用的章节，上篇提到的熔断能力，是历年保障大促当天晚上整个系统不被洪峰流量打垮的法宝。本文将重点介绍为什么我们要做混沌工程以及如何使用 ChaoBlade 工具和 AHAS 平台快速实施混沌工程。</p>
<p><strong>前言</strong></p>
<p>从上篇开始，我们进入到了高可用的章节，上篇提到的熔断能力，是历年保障大促当天晚上整个系统不被洪峰流量打垮的法宝，本篇介绍的措施与熔断有不一样的地方，一个是线上洪峰来临时的保护措施，它更多的是流量低峰或者在专门的演练环境中，针对可能遇见的各类故障，采取演练的手段，来窥探对业务的影响；它的主要目的是让我们自己更加了解自己业务系统的薄弱环节，以便来对症下药增强系统的高可用能力。</p>
<p><strong>为什么需要混沌工程？</strong></p>
<p>任何一个系统都会有未曾可知的故障出现，拿现代工艺已经很好的磁盘来说，有统计数据的磁盘最低的年故障率都可达到 0.39% 。即便是这么底层基础设施，也会有这么高的不确定性。</p>
<p>尤其当下大部分的服务形态都是分布式架构，在分布式系统架构下，服务间的依赖日益复杂，更很难评估单个服务故障对整个系统的影响；并且请求链路长，监控告警的不完善导致发现问题、定位问题难度增大；同时业务和技术迭代快，如何持续保障系统的稳定性和高可用性受到很大的挑战。</p>
<h3 id=""><a href="#" class="headerlink" title=""></a></h3><h3 id="1-云原生系统挑战更大"><a href="#1-云原生系统挑战更大" class="headerlink" title="1. 云原生系统挑战更大"></a><strong>1. 云原生系统挑战更大</strong></h3><p>谈到云原生，可以说云原生是一个理念，主要包含的技术有云设施、容器、微服务、服务网格、Serverless 等技术。云设施指公有云、专有云和混合云等，是云原生系统的基础设施，基础实施的故障可能对整个上层业务系统造成很大影响，所以说云设施的稳定性是非常重要的。</p>
<p><strong>容器服务的挑战</strong>可以分两大类：一类是面向 K8s 服务提供商，服务是否稳定；另一类是面向用户，配置的扩缩容规则是否有效，实现的 CRD 是否正确，容器编排是否合理等问题。</p>
<p><strong>分布式服务的挑战</strong>主要是复杂性，单个服务的故障很难判断对整个系统的影响；service mesh，sidecar 的服务路由、负载均衡等功能的有效性，还有 sidecar 容器本身的可用性。</p>
<p><strong>一些新兴的部署模式的挑战</strong>如 serverless，现在基本上都是函数加事件的形式，资源调度是否有效，而且 serverless 服务提供商屏蔽了一些中间件，你能掌控的是函数这些服务，那么你可以通过混沌工程去验证你函数调用的一些配置，比如超时配置、相关的一些降级策略等这些是否合理。</p>
<p>以上技术都有相同的共性，比如弹性可扩展、松耦合、容错性高、还有一些易于管理，便于观察这些特性。所以说在云原生时代，通过混沌工程可以更有效的推进系统的“云原生”化。</p>
<h3 id="-1"><a href="#-1" class="headerlink" title=""></a></h3><h3 id="2-每个职位都需要懂混沌工程"><a href="#2-每个职位都需要懂混沌工程" class="headerlink" title="2. 每个职位都需要懂混沌工程"></a><strong>2. 每个职位都需要懂混沌工程</strong></h3><p>混沌工程是一种思想，它让系统中的每个参与者都学会去考虑一件事情：如果所依赖的某服务中断了服务该怎么办？对于以下四类人群而言，意义尤显突出：</p>
<ul>
<li><p>对于<strong>架构师</strong>来说，可以验证系统架构的容错能力，我们需要面向失败设计的系统，混沌工程的思想就是践行这一原则的方式；</p>
</li>
<li><p>对于<strong>开发和运维</strong>，可以提高故障的应急效率，实现故障告警、定位、恢复的有效和高效性；</p>
</li>
<li><p>对于<strong>测试</strong>来说，可以弥补传统测试方法留下的空白，之前的测试方法基本上是从用户的角度去做，而混沌工程是从系统的角度进行测试，降低故障复发率；</p>
</li>
<li><p>对于<strong>产品和设计</strong>，通过混沌事件查看产品的表现，提升客户使用体验。所以说混沌工程面向的不仅仅是开发、测试，拥有最好的客户体验是每个人的目标，所以实施混沌工程，可以提早发现生产环境上的问题，并且可以以战养战，提升故障应急效率和可以使用体验，逐渐建设高可用的韧性系统。</p>
</li>
</ul>
<h2 id="-2"><a href="#-2" class="headerlink" title=""></a></h2><p><img src="http://iubest.gitee.io/pic/640-1601366814197.gif" alt="动态黑色音符"></p>
<p><strong>混沌工程实操</strong></p>
<p>在一次完整的演练流程中，需要先做好计划，对相关的演练计划有一个行为预期；演练相关计划的同时，我们推荐的最佳实践是需要配合有业务的自动化测试，每演练一次需要全方位的跑完自动化测试用例，这样才能全面的了解真正的业务产生时对业务造成的影响：</p>
<p><img src="http://iubest.gitee.io/pic/640-1601003235339.webp" alt="img"></p>
<p>在上面的图中描述了一次完整的故障演练需要经过的步骤，其中最重要的一步的实践是如何“执行预制混沌实验”？因为这一步需要一个专业的工具，在业内目前最流行的工具是 Netflix 的 Chaos Monkey 和阿里巴巴开源的<a target="_blank" rel="noopener" href="https://mp.weixin.qq.com/s?__biz=MzUzNzYxNjAzMg==&mid=2247494460&idx=2&sn=2d50c4257bf7ee4f780255923c3f4c51&chksm=fae6e0f3cd9169e56378a5c85ca54a1eda4e607bd339192be19d99774889fc1818f554f925db&mpshare=1&scene=24&srcid=0921mEuiYl0Z71dEPujp5MTa&sharer_sharetime=1600732341212&sharer_shareid=407c90840c4caeeaf9680b1dd38c62ba&key=dbdefc5d690db19540edcfe32db55dc38284f6437256b923bdcadaf128424ac3eae8a68d9b758ebba248a653a1c83d17a477d058632881c7d16d97c3fb7f634a4d52cf825bc6d8f947b9324e02a21966c631338d2914dd8be97b712780e5cd364120a93d1124246963c52e2b5217cf5f0ad70b9dbd5eafb48cd6732c1077cbf5&ascene=14&uin=MTIwMjI3NTkwNQ==&devicetype=Windows+10+x64&version=62090529&lang=zh_CN&exportkey=A8Zk0u8xHdVwPzIwFzJfv6c=&pass_ticket=pdKql0fF0rGOXvNr/tToA1+AardNoo77GWcTcNS7PpaVOYI2W/vk8qbSO4P5qmER&wx_header=0&winzoom=1">ChaosBlade</a>，我们接下来主要是介绍如何使用 ChaosBlade 来完成一次演练。</p>
<h3 id="1-使用-ChaosBlade-去做"><a href="#1-使用-ChaosBlade-去做" class="headerlink" title="1. 使用 ChaosBlade 去做"></a><strong>1. 使用 ChaosBlade 去做</strong></h3><p><a target="_blank" rel="noopener" href="https://mp.weixin.qq.com/s?__biz=MzUzNzYxNjAzMg==&mid=2247494460&idx=2&sn=2d50c4257bf7ee4f780255923c3f4c51&chksm=fae6e0f3cd9169e56378a5c85ca54a1eda4e607bd339192be19d99774889fc1818f554f925db&mpshare=1&scene=24&srcid=0921mEuiYl0Z71dEPujp5MTa&sharer_sharetime=1600732341212&sharer_shareid=407c90840c4caeeaf9680b1dd38c62ba&key=dbdefc5d690db19540edcfe32db55dc38284f6437256b923bdcadaf128424ac3eae8a68d9b758ebba248a653a1c83d17a477d058632881c7d16d97c3fb7f634a4d52cf825bc6d8f947b9324e02a21966c631338d2914dd8be97b712780e5cd364120a93d1124246963c52e2b5217cf5f0ad70b9dbd5eafb48cd6732c1077cbf5&ascene=14&uin=MTIwMjI3NTkwNQ==&devicetype=Windows+10+x64&version=62090529&lang=zh_CN&exportkey=A8Zk0u8xHdVwPzIwFzJfv6c=&pass_ticket=pdKql0fF0rGOXvNr/tToA1+AardNoo77GWcTcNS7PpaVOYI2W/vk8qbSO4P5qmER&wx_header=0&winzoom=1">ChaosBlade</a> 是阿里巴巴一款遵循混沌实验模型的混沌实验执行工具，具有场景丰富度高，简单易用等特点，而且扩展场景也特别方便，开源不久就被加入到 CNCF Landspace 中，成为主流的一款混沌工具。目前包含的场景有基础资源、应用服务、容器服务、云资源等。ChaosBlade 下载解压即用，可以通过执行 blade 命令来执行云原生下微服务的演练场景，下面是模拟 Kubernetes 下微服务中数据库调用延迟故障。</p>
<p><img src="http://iubest.gitee.io/pic/640-1601003235283.webp" alt="img"></p>
<h3 id="-3"><a href="#-3" class="headerlink" title=""></a></h3><h3 id="2-使用-AHAS-故障演练平台去做"><a href="#2-使用-AHAS-故障演练平台去做" class="headerlink" title="2. 使用 AHAS 故障演练平台去做"></a><strong>2. 使用 AHAS 故障演练平台去做</strong></h3><p>AHAS 故障演练平台是阿里云对外部用户开放的云产品，使用方式可参考<a target="_blank" rel="noopener" href="https://mp.weixin.qq.com/s?__biz=MzUzNzYxNjAzMg==&mid=2247494460&idx=2&sn=2d50c4257bf7ee4f780255923c3f4c51&chksm=fae6e0f3cd9169e56378a5c85ca54a1eda4e607bd339192be19d99774889fc1818f554f925db&mpshare=1&scene=24&srcid=0921mEuiYl0Z71dEPujp5MTa&sharer_sharetime=1600732341212&sharer_shareid=407c90840c4caeeaf9680b1dd38c62ba&key=dbdefc5d690db19540edcfe32db55dc38284f6437256b923bdcadaf128424ac3eae8a68d9b758ebba248a653a1c83d17a477d058632881c7d16d97c3fb7f634a4d52cf825bc6d8f947b9324e02a21966c631338d2914dd8be97b712780e5cd364120a93d1124246963c52e2b5217cf5f0ad70b9dbd5eafb48cd6732c1077cbf5&ascene=14&uin=MTIwMjI3NTkwNQ==&devicetype=Windows+10+x64&version=62090529&lang=zh_CN&exportkey=A8Zk0u8xHdVwPzIwFzJfv6c=&pass_ticket=pdKql0fF0rGOXvNr/tToA1+AardNoo77GWcTcNS7PpaVOYI2W/vk8qbSO4P5qmER&wx_header=0&winzoom=1">官方文档</a>。其底层的故障注入能力大部分来源于 ChaosBlade 实现，另一部分使用自身小程序扩展实现。AHAS 相比于 ChaosBlade，除了简单易用的白屏操作之外，还实现了上层的演练编排、权限控制、场景管理等，而且还针对微服务新增应用维度演练，简化演练成本，优化演练体验。</p>
<p><img src="http://iubest.gitee.io/pic/640-1601003235331.webp" alt="img"></p>
<p><img src="http://iubest.gitee.io/pic/640-1601366806598.gif" alt="动态黑色音符"></p>
<p><strong>结尾</strong></p>
<p>混沌工程是一种主动防御的稳定性手段，体现的是反脆弱的思想，实施混沌工程不能只是把故障制造出来，需要有明确的驱动目标。我们要选择合适的工具和平台，控制演练风险，实现常态化演练。</p>
<p>阿里巴巴内部从最早引入混沌工程解决微服务的依赖问题，到业务服务、云服务稳态验证，进一步升级到公共云、专有云的业务连续性保障，以及在验证云原生系统的稳定性等方面积累了比较丰富的场景和实践经验；这一些经验沉淀我们都通过开源产品以及云产品<a target="_blank" rel="noopener" href="https://mp.weixin.qq.com/s?__biz=MzUzNzYxNjAzMg==&mid=2247494460&idx=2&sn=2d50c4257bf7ee4f780255923c3f4c51&chksm=fae6e0f3cd9169e56378a5c85ca54a1eda4e607bd339192be19d99774889fc1818f554f925db&mpshare=1&scene=24&srcid=0921mEuiYl0Z71dEPujp5MTa&sharer_sharetime=1600732341212&sharer_shareid=407c90840c4caeeaf9680b1dd38c62ba&key=dbdefc5d690db19540edcfe32db55dc38284f6437256b923bdcadaf128424ac3eae8a68d9b758ebba248a653a1c83d17a477d058632881c7d16d97c3fb7f634a4d52cf825bc6d8f947b9324e02a21966c631338d2914dd8be97b712780e5cd364120a93d1124246963c52e2b5217cf5f0ad70b9dbd5eafb48cd6732c1077cbf5&ascene=14&uin=MTIwMjI3NTkwNQ==&devicetype=Windows+10+x64&version=62090529&lang=zh_CN&exportkey=A8Zk0u8xHdVwPzIwFzJfv6c=&pass_ticket=pdKql0fF0rGOXvNr/tToA1+AardNoo77GWcTcNS7PpaVOYI2W/vk8qbSO4P5qmER&wx_header=0&winzoom=1">AHAS</a> 一一对外输出。</p>
 
      <!-- reward -->
      
      <div id="reword-out">
        <div id="reward-btn">
          打赏
        </div>
      </div>
      
    </div>
    

    <!-- copyright -->
    
    <div class="declare">
      <ul class="post-copyright">
        <li>
          <i class="ri-copyright-line"></i>
          <strong>版权声明： </strong>
          
          本博客所有文章除特别声明外，著作权归作者所有。转载请注明出处！
          
        </li>
      </ul>
    </div>
    
    <footer class="article-footer">
       
<div class="share-btn">
      <span class="share-sns share-outer">
        <i class="ri-share-forward-line"></i>
        分享
      </span>
      <div class="share-wrap">
        <i class="arrow"></i>
        <div class="share-icons">
          
          <a class="weibo share-sns" href="javascript:;" data-type="weibo">
            <i class="ri-weibo-fill"></i>
          </a>
          <a class="weixin share-sns wxFab" href="javascript:;" data-type="weixin">
            <i class="ri-wechat-fill"></i>
          </a>
          <a class="qq share-sns" href="javascript:;" data-type="qq">
            <i class="ri-qq-fill"></i>
          </a>
          <a class="douban share-sns" href="javascript:;" data-type="douban">
            <i class="ri-douban-line"></i>
          </a>
          <!-- <a class="qzone share-sns" href="javascript:;" data-type="qzone">
            <i class="icon icon-qzone"></i>
          </a> -->
          
          <a class="facebook share-sns" href="javascript:;" data-type="facebook">
            <i class="ri-facebook-circle-fill"></i>
          </a>
          <a class="twitter share-sns" href="javascript:;" data-type="twitter">
            <i class="ri-twitter-fill"></i>
          </a>
          <a class="google share-sns" href="javascript:;" data-type="google">
            <i class="ri-google-fill"></i>
          </a>
        </div>
      </div>
</div>

<div class="wx-share-modal">
    <a class="modal-close" href="javascript:;"><i class="ri-close-circle-line"></i></a>
    <p>扫一扫，分享到微信</p>
    <div class="wx-qrcode">
      <img src="//api.qrserver.com/v1/create-qr-code/?size=150x150&data=http://example.com/2020/11/11/k8s/Spring%20Cloud%20%E5%BA%94%E7%94%A8%E5%9C%A8%20Kubernetes%20%E4%B8%8A%E7%9A%84%E6%9C%80%E4%BD%B3%E5%AE%9E%E8%B7%B5%20%E2%80%94%20%E9%AB%98%E5%8F%AF%E7%94%A8%20%E6%B7%B7%E6%B2%8C%E5%B7%A5%E7%A8%8B/" alt="微信分享二维码">
    </div>
</div>

<div id="share-mask"></div>  
  <ul class="article-tag-list" itemprop="keywords"><li class="article-tag-list-item"><a class="article-tag-list-link" href="/tags/k8s/" rel="tag">k8s</a></li></ul>

    </footer>
  </div>

   
  <nav class="article-nav">
    
      <a href="/2020/11/11/k8s/Kubernetes%20%E6%95%85%E9%9A%9C%E8%A7%A3%E5%86%B3%E5%BF%83%E5%BE%97/" class="article-nav-link">
        <strong class="article-nav-caption">上一篇</strong>
        <div class="article-nav-title">
          
            Kubernetes 故障解决心得.md
          
        </div>
      </a>
    
    
      <a href="/2020/11/11/k8s/%E4%B8%80%E6%96%87%E8%AE%B2%E6%B8%85%E6%A5%9ACeph%E5%88%86%E5%B8%83%E5%BC%8F%E5%AD%98%E5%82%A8/" class="article-nav-link">
        <strong class="article-nav-caption">下一篇</strong>
        <div class="article-nav-title">一文讲清楚Ceph分布式存储.md</div>
      </a>
    
  </nav>

   
<!-- valine评论 -->
<div id="vcomments-box">
  <div id="vcomments"></div>
</div>
<script src="//cdn1.lncld.net/static/js/3.0.4/av-min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/valine@1.4.14/dist/Valine.min.js"></script>
<script>
  new Valine({
    el: "#vcomments",
    app_id: "",
    app_key: "",
    path: window.location.pathname,
    avatar: "monsterid",
    placeholder: "给我的文章加点评论吧~",
    recordIP: true,
  });
  const infoEle = document.querySelector("#vcomments .info");
  if (infoEle && infoEle.childNodes && infoEle.childNodes.length > 0) {
    infoEle.childNodes.forEach(function (item) {
      item.parentNode.removeChild(item);
    });
  }
</script>
<style>
  #vcomments-box {
    padding: 5px 30px;
  }

  @media screen and (max-width: 800px) {
    #vcomments-box {
      padding: 5px 0px;
    }
  }

  #vcomments-box #vcomments {
    background-color: #fff;
  }

  .v .vlist .vcard .vh {
    padding-right: 20px;
  }

  .v .vlist .vcard {
    padding-left: 10px;
  }
</style>

 
     
</article>

</section>
      <footer class="footer">
  <div class="outer">
    <ul>
      <li>
        Copyrights &copy;
        2015-2020
        <i class="ri-heart-fill heart_icon"></i> TzWind
      </li>
    </ul>
    <ul>
      <li>
        
        
        
        由 <a href="https://hexo.io" target="_blank">Hexo</a> 强力驱动
        <span class="division">|</span>
        主题 - <a href="https://github.com/Shen-Yu/hexo-theme-ayer" target="_blank">Ayer</a>
        
      </li>
    </ul>
    <ul>
      <li>
        
        
        <span>
  <span><i class="ri-user-3-fill"></i>访问人数:<span id="busuanzi_value_site_uv"></span></s>
  <span class="division">|</span>
  <span><i class="ri-eye-fill"></i>浏览次数:<span id="busuanzi_value_page_pv"></span></span>
</span>
        
      </li>
    </ul>
    <ul>
      
    </ul>
    <ul>
      
    </ul>
    <ul>
      <li>
        <!-- cnzz统计 -->
        
        <script type="text/javascript" src='https://s9.cnzz.com/z_stat.php?id=1278069914&amp;web_id=1278069914'></script>
        
      </li>
    </ul>
  </div>
</footer>
      <div class="float_btns">
        <div class="totop" id="totop">
  <i class="ri-arrow-up-line"></i>
</div>

<div class="todark" id="todark">
  <i class="ri-moon-line"></i>
</div>

      </div>
    </main>
    <aside class="sidebar on">
      <button class="navbar-toggle"></button>
<nav class="navbar">
  
  <div class="logo">
    <a href="/"><img src="/images/ayer-side.svg" alt="Hexo"></a>
  </div>
  
  <ul class="nav nav-main">
    
    <li class="nav-item">
      <a class="nav-item-link" href="/">主页</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" href="/archives">归档</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" href="/categories">分类</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" href="/tags">标签</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" target="_blank" rel="noopener" href="http://www.baidu.com">百度</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" href="/friends">友链</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" href="/2019/about">关于我</a>
    </li>
    
  </ul>
</nav>
<nav class="navbar navbar-bottom">
  <ul class="nav">
    <li class="nav-item">
      
      <a class="nav-item-link nav-item-search"  title="搜索">
        <i class="ri-search-line"></i>
      </a>
      
      
      <a class="nav-item-link" target="_blank" href="/atom.xml" title="RSS Feed">
        <i class="ri-rss-line"></i>
      </a>
      
    </li>
  </ul>
</nav>
<div class="search-form-wrap">
  <div class="local-search local-search-plugin">
  <input type="search" id="local-search-input" class="local-search-input" placeholder="Search...">
  <div id="local-search-result" class="local-search-result"></div>
</div>
</div>
    </aside>
    <script>
      if (window.matchMedia("(max-width: 768px)").matches) {
        document.querySelector('.content').classList.remove('on');
        document.querySelector('.sidebar').classList.remove('on');
      }
    </script>
    <div id="mask"></div>

<!-- #reward -->
<div id="reward">
  <span class="close"><i class="ri-close-line"></i></span>
  <p class="reward-p"><i class="ri-cup-line"></i>请我喝杯咖啡吧~</p>
  <div class="reward-box">
    
    
  </div>
</div>
    
<script src="/js/jquery-2.0.3.min.js"></script>


<script src="/js/lazyload.min.js"></script>

<!-- Tocbot -->


<script src="/js/tocbot.min.js"></script>

<script>
  tocbot.init({
    tocSelector: '.tocbot',
    contentSelector: '.article-entry',
    headingSelector: 'h1, h2, h3, h4, h5, h6',
    hasInnerContainers: true,
    scrollSmooth: true,
    scrollContainer: 'main',
    positionFixedSelector: '.tocbot',
    positionFixedClass: 'is-position-fixed',
    fixedSidebarOffset: 'auto'
  });
</script>

<script src="https://cdn.jsdelivr.net/npm/jquery-modal@0.9.2/jquery.modal.min.js"></script>
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/jquery-modal@0.9.2/jquery.modal.min.css">
<script src="https://cdn.jsdelivr.net/npm/justifiedGallery@3.7.0/dist/js/jquery.justifiedGallery.min.js"></script>

<script src="/dist/main.js"></script>

<!-- ImageViewer -->

<!-- Root element of PhotoSwipe. Must have class pswp. -->
<div class="pswp" tabindex="-1" role="dialog" aria-hidden="true">

    <!-- Background of PhotoSwipe. 
         It's a separate element as animating opacity is faster than rgba(). -->
    <div class="pswp__bg"></div>

    <!-- Slides wrapper with overflow:hidden. -->
    <div class="pswp__scroll-wrap">

        <!-- Container that holds slides. 
            PhotoSwipe keeps only 3 of them in the DOM to save memory.
            Don't modify these 3 pswp__item elements, data is added later on. -->
        <div class="pswp__container">
            <div class="pswp__item"></div>
            <div class="pswp__item"></div>
            <div class="pswp__item"></div>
        </div>

        <!-- Default (PhotoSwipeUI_Default) interface on top of sliding area. Can be changed. -->
        <div class="pswp__ui pswp__ui--hidden">

            <div class="pswp__top-bar">

                <!--  Controls are self-explanatory. Order can be changed. -->

                <div class="pswp__counter"></div>

                <button class="pswp__button pswp__button--close" title="Close (Esc)"></button>

                <button class="pswp__button pswp__button--share" style="display:none" title="Share"></button>

                <button class="pswp__button pswp__button--fs" title="Toggle fullscreen"></button>

                <button class="pswp__button pswp__button--zoom" title="Zoom in/out"></button>

                <!-- Preloader demo http://codepen.io/dimsemenov/pen/yyBWoR -->
                <!-- element will get class pswp__preloader--active when preloader is running -->
                <div class="pswp__preloader">
                    <div class="pswp__preloader__icn">
                        <div class="pswp__preloader__cut">
                            <div class="pswp__preloader__donut"></div>
                        </div>
                    </div>
                </div>
            </div>

            <div class="pswp__share-modal pswp__share-modal--hidden pswp__single-tap">
                <div class="pswp__share-tooltip"></div>
            </div>

            <button class="pswp__button pswp__button--arrow--left" title="Previous (arrow left)">
            </button>

            <button class="pswp__button pswp__button--arrow--right" title="Next (arrow right)">
            </button>

            <div class="pswp__caption">
                <div class="pswp__caption__center"></div>
            </div>

        </div>

    </div>

</div>

<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/photoswipe@4.1.3/dist/photoswipe.min.css">
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/photoswipe@4.1.3/dist/default-skin/default-skin.min.css">
<script src="https://cdn.jsdelivr.net/npm/photoswipe@4.1.3/dist/photoswipe.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/photoswipe@4.1.3/dist/photoswipe-ui-default.min.js"></script>

<script>
    function viewer_init() {
        let pswpElement = document.querySelectorAll('.pswp')[0];
        let $imgArr = document.querySelectorAll(('.article-entry img:not(.reward-img)'))

        $imgArr.forEach(($em, i) => {
            $em.onclick = () => {
                // slider展开状态
                // todo: 这样不好，后面改成状态
                if (document.querySelector('.left-col.show')) return
                let items = []
                $imgArr.forEach(($em2, i2) => {
                    let img = $em2.getAttribute('data-idx', i2)
                    let src = $em2.getAttribute('data-target') || $em2.getAttribute('src')
                    let title = $em2.getAttribute('alt')
                    // 获得原图尺寸
                    const image = new Image()
                    image.src = src
                    items.push({
                        src: src,
                        w: image.width || $em2.width,
                        h: image.height || $em2.height,
                        title: title
                    })
                })
                var gallery = new PhotoSwipe(pswpElement, PhotoSwipeUI_Default, items, {
                    index: parseInt(i)
                });
                gallery.init()
            }
        })
    }
    viewer_init()
</script>

<!-- MathJax -->

<!-- Katex -->

<!-- busuanzi  -->


<script src="/js/busuanzi-2.3.pure.min.js"></script>


<!-- ClickLove -->

<!-- ClickBoom1 -->

<!-- ClickBoom2 -->

<!-- CodeCopy -->


<link rel="stylesheet" href="/css/clipboard.css">

<script src="https://cdn.jsdelivr.net/npm/clipboard@2/dist/clipboard.min.js"></script>
<script>
  function wait(callback, seconds) {
    var timelag = null;
    timelag = window.setTimeout(callback, seconds);
  }
  !function (e, t, a) {
    var initCopyCode = function(){
      var copyHtml = '';
      copyHtml += '<button class="btn-copy" data-clipboard-snippet="">';
      copyHtml += '<i class="ri-file-copy-2-line"></i><span>COPY</span>';
      copyHtml += '</button>';
      $(".highlight .code pre").before(copyHtml);
      $(".article pre code").before(copyHtml);
      var clipboard = new ClipboardJS('.btn-copy', {
        target: function(trigger) {
          return trigger.nextElementSibling;
        }
      });
      clipboard.on('success', function(e) {
        let $btn = $(e.trigger);
        $btn.addClass('copied');
        let $icon = $($btn.find('i'));
        $icon.removeClass('ri-file-copy-2-line');
        $icon.addClass('ri-checkbox-circle-line');
        let $span = $($btn.find('span'));
        $span[0].innerText = 'COPIED';
        
        wait(function () { // 等待两秒钟后恢复
          $icon.removeClass('ri-checkbox-circle-line');
          $icon.addClass('ri-file-copy-2-line');
          $span[0].innerText = 'COPY';
        }, 2000);
      });
      clipboard.on('error', function(e) {
        e.clearSelection();
        let $btn = $(e.trigger);
        $btn.addClass('copy-failed');
        let $icon = $($btn.find('i'));
        $icon.removeClass('ri-file-copy-2-line');
        $icon.addClass('ri-time-line');
        let $span = $($btn.find('span'));
        $span[0].innerText = 'COPY FAILED';
        
        wait(function () { // 等待两秒钟后恢复
          $icon.removeClass('ri-time-line');
          $icon.addClass('ri-file-copy-2-line');
          $span[0].innerText = 'COPY';
        }, 2000);
      });
    }
    initCopyCode();
  }(window, document);
</script>


<!-- CanvasBackground -->


    
  </div>
</body>

</html>